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Judging Student Achievement 

Why Getting the Right Data Matters 


S tates are engaged in serious efforts to raise student achievement, 
paying particular attention to narrowing the achievement gap 
between high- and low-performing students that has persisted 
in our public schools for decades. The achievement gap is particularly 
troubling because, all too often, those at low achievement levels are 
minority, special education, or limited-English-proficient students. 

The federal No Child Left Behind (NCLB) act has given added impetus 
to efforts to close the achievement gap. States are now testing all 
students in specific grades in mathematics and reading, using stan- 
dardized state assessments aligned with state academic standards. 
Statewide, district, and school test scores must be disaggregated to 
show results for various groups of students, so that average test results 
cannot hide continuing lower achievement by specific student sub- 
groups. Schools failing to make adequate yearly progress over time 
face increasingly strict sanctions. 

Efforts to close the achievement gap are as diverse as the 50 states, 
and many educators are struggling to implement school improvement 
strategies and find adequate and appropriate measurements to see if 
these strategies are working. NCLB offers a useful framework for think- 
ing about the kinds of data needed to identify students having dif- 
ficulty meeting standards and to document school progress in closing 
the achievement gap over time. 





THE DATA WE HAVE NOW 

Cross-sectional data are the norm in states. 

At present, most states rely on cross-sectional data to doc- 
ument changes in academic achievement among students 
and schools. Cross-sectional data produce a "snapshot" of 
academic achievement at a fixed point in time and provide 
important information on a variety of school and student 
outcomes, including achievement gaps among student 
subgroups in a school, district, or state. Compared to 
other types of data, cross-sectional data are also fairly easy 
to collect and inexpensive to maintain. 


that will permit matching of individual student records 
over time and across education databases. The advent of 
NCLB has made this need even more imperative. 

BENEFITS OF LONGITUDINAL DATA 

In contrast to cross-sectional data, longitudinal data can 
provide a wealth of information to help schools and dis- 
tricts assess student performance over time and to moni- 
tor progress in closing the achievement gap. Truly useful 
education data should be able to answer, at minimum, a 
series of critical policy and evaluation questions, including 
those related to: 


One significant limitation of cross-sectional data is that they 
do not track individual students over time. For example, 
cross-sectional data cannot reveal whether students are ac- 
tually improving their academic skills as they move through 
school, nor do they account for vital information, such as 
prior student achievement, that can help educators make 
diagnostic decisions about which students need which kind 
of help. Without longitudinal data, educators and policy- 
makers lack valuable information that can help them chart 
the most promising course to improving student learning. 

THE DATA WE NEED (NOW) 


Assessing individual student academic growth 
and proficiency over time, the best measure of 
schools' impact on student achievement. 

Monitoring student mobility, retention, and attri- 
tion, all factors that are related to student achievement 
and school performance. 

Examining prior achievement for all student sub- 
groups, since prior achievement is a solid predictor of 
future performance. 

Predicting future student achievement, so that edu- 
cators can determine whether they are adequately pre- 
paring students for future success. 


Longitudinal data permit 
powerful analyses of student 
achievement. 

To obtain the most accurate and 
useful data, state data systems 
will need to be reorganized to 
gather longitudinal data for indi- 
vidual students. States will need 
to develop and maintain central- 
ized, longitudinal databases and 
assign unique statewide identi- 
fiers (such as a Social Security 
number or other code) to children 


What the right data can 
help us do 

The right data make it possible to: 

• Evaluate the effectiveness of schools 
and programs in improving student 
achievement. 

• Identify consistently high-performing 
schools so that educators can learn 
about promising practices. 

• Intervene in a timely way to help stu- 
dents who are struggling. 

• Determine how well schools are prepar- 
ing students to complete high school 
and enroll in postsecondary education. 


With longitudinal student data, 
educators and policymakers can 
better understand how their stu- 
dents are performing over time 
and how their schools are doing 
in addressing the academic needs 
of all students. The next section 
provides answers to specific ex- 
amples of the types of questions 
that longitudinal student data 
can address. 



Judging Student Achievement • September 2005 2 


ASSESSING STUDENT ACADEMIC GROWTH AND PROFICIENCY OVER TIME 




Sample questions that longitudinal data can 
answer: 

• What percentage of last year's less- 
than-proficient students achieved 
more than a year's growth in academic 
achievement this year? 

• Are students who have been taught 
the new mathematics curriculum im- 
proving their achievement on the state 
assessment over time? 

• Are this year's achievement gaps nar- 
rower than would have been predicted 
by last year's achievement gaps for the 
same students? 

Using longitudinal data, educators can track 
changes in the achievement gap for specific 
groups of students as they progress through 
school. This permits more accurate measure- 
ment of the impact of schools in closing the 
achievement gap — for example, one group of 
English language learners may have started 
out behind native English speakers, but the 
school is making exceptional progress in help- 
ing them catch up. 


Figure 1 documents how tracking a single co- 
hort of students over time can provide more 
reliable data than comparing performances 
for different populations of students. With- 
out longitudinal data on student outcomes, 
comparing different populations of students 
may give the appearance that students are 
backsliding, even when performance differ- 
ences result from external factors, when in 
fact performance is actually improving. 

Figure 1 

Percentage of students meeting or exceeding state 
mathematics standards, cohort vs. grade level perfor- 
mance: 2002 and 2004 

Percent The same 8th- 



Cohort 

+ Cross-sectional 
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MONITORING STUDENT MOBILITY, RETENTION, AND ATTRITION 


Additional sample questions that longitudinal 
data can answer: 

• What percentage of at-risk 7th-graders 
subsequently dropped out? 

• What percentage of 9th-grade stu- 
dents participating in remedial reading 
programs obtained their high school 
diploma? 

• How many of the highest performing 
students are leaving the lowest per- 
forming schools? 

With longitudinal student data, states can 
identify students who leave and enroll in 
another public school in the state, as well as 
those who leave the state's public education 
system. With this information, states can get 
a more accurate picture of dropout rates. For 
example, by taking the average percentage of 
students who leave public schools in grades 
1 through 7, the state can establish a bench- 
mark for "normal" attrition. Attrition in excess 
of this ("excess attrition") can be viewed as a 
rough estimate of the dropout rate in later 
grades, as shown in Figure 2. 

As shown, the dropout rate of students who 
were in grade 9 in the prior year is likely to be 


at least 6 percent (1 1 percent attrition for that 
year, minus approximately 5 percent normal 
attrition). 

Attrition can also affect high school test 
scores. If the lowest performing students drop 
out, a school with a higher dropout rate might 
show improvement in state test scores based 
on the proficiency levels of the remaining stu- 
dents. Consequently, it is not possible to assess 
the relationship between school-level attrition 
and high school proficiency rates on state tests 
without longitudinal student-level data. 

Figure 2 

Attrition: Statewide percentage of students en- 
rolled in 2002 but not enrolled in 2003 

Percent 



10 11 12 

Grade level 

-■■Year-to-year attrition rate 
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EXAMINING PRIOR ACHIEVEMENT FOR ALL STUDENT SUBGROUPS 


Other questions that we can use longitudinal 
data to answer: 

• How many 8th-grade proficient and 
advanced students passed at least one 
Advanced Placement (AP) exam in high 
school? 

• How many 8th-grade less-than-pro- 
ficient students graduated from high 
school in four years? 

• Which instructional programs in mid- 
dle school prepare low-performing stu- 
dents for the high school curriculum? 

Since academic knowledge is cumulative, 
students' past performance can usually pre- 
dict future results. Unique student identifiers 
can track students' performance retrospec- 
tively to see if there is a pattern over time. 
For example, student test scores below state 
thresholds in the current year may be the 
result, in part, of skill deficits apparent years 
earlier but never remedied. With longitudinal 
data in hand, school administrators can iden- 
tify earlier grades or specific programs where 
educational intervention might be needed. 


Longitudinal data can also be used to assess 
whether certain students, such as those who 
attended a specific feeder school, are affect- 
ing school-wide or subgroup performances. 
As Figure 3 illustrates, although school-wide 
test results may exceed state standards, more 
detailed analysis reveals that students' test 
scores are at least partly a function of the 
middle school they attended. 


Figure 3 

Percentage of lOth-grade students meeting or 
exceeding state mathematics standards, state- 
wide and by feeder school: 2004 

Percent 

100 



Statewide School Feeder Feeder 
middle middle 
school A school B 

■ Advanced 

■ Proficient 
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Longitudinal data can also provide answers to 
these questions: 

• How does participation in advanced high 
school courses affect college success? 

• Do remedial reading programs im- 
prove students' chances of passing 
exit exams required for graduation? 

• Are specific school characteristics, such 
as instructional program, teacher qual- 
ifications, size, or attendance rates 
associated with success in narrowing 
achievement gaps over time? 

Linking student performance over time can 
enable educators to make cause-and-effect 
connections between students' prior abilities 
and subsequent academic performance. This 
information can be used to identify students 
at high risk of school failure, enabling school 
administrators to target resources more effec- 
tively to those in greatest need. 

Prior achievement is a powerful indicator of 
which students are most at risk, and a high 
school's performance with students who 
entered with low 8th-grade scores is a good 
indicator of the school's performance with 
at-risk students. For example, Figure 4 shows 
that students who achieved mathematics 
proficiency in the 8th grade were much more 
likely to pass 1 1 th-grade mathematics. The 
figure also shows a much wider achievement 
gap for low-income students who were below 
proficient in the 8th grade, indicating that 
the school may need to sharpen its focus on 


intensive mathematics instruction for low-per- 
forming, low-income students. 

Other predictive measures that can provide 
a more nuanced view of performance include 
examining the relationship of student test 
scores to their completion rates for state- 
required or AP coursework or the SAT or ACT 
exams. 

State administrators can also use student 
identifiers to track graduates once they 
leave the public school system and enroll in 
a state postsecondary institution. States us- 
ing a student's Social Security number as a 
unique identifier will be able to access state 
unemployment insurance (Ul) wage records to 
determine if there is a link between school 
achievement and subsequent labor market 
success. 

Figure 4 

Percentage of students passing 11 th-grade 
mathematics, by 8th-grade mathematics profi- 
ciency and income 

Percent 



in group were proficient below proficiency 
in 8th grade in 8th grade 

■ Non-low-income students ■ Low-income students 
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OTHER ADVANTAGES OF 
COMPREHENSIVE STUDENT 
DATA SYSTEMS 

Besides enabling states to fulfill NCLB requirements, a com- 
prehensive student data system offers other advantages: 

Consistent data definitions. Consistent definitions make 
it possible to analyze an issue across databases, districts, 
and even potentially across different states. 

Consolidation of data. Once a comprehensive student 
data system is in place, it should be possible to eliminate 
redundancy in data collection across state, district, and lo- 
cal databases. 

Flexibility to meet changing federal and state re- 
quirements. As states and the federal government 
monitor student achievement, changes may be made in 
programs, regulations, or other requirements that neces- 
sitate new data or analyses. 

Improved evaluation of programs and better data 
quality. Adding other measures to cross-sectional data 
collection will enable states to pinpoint more precisely how 
well students are doing and to find optimal points of inter- 
vention in the education process. Such data will also pro- 
vide a more reliable and valid basis for policy decisions. 1 

WHAT INFORMATION IS NEEDED FOR A 
COMPREHENSIVE DATA SYSTEM? 

A comprehensive student data collection system must start 
with a unique student identifier, such as a Social Security 
number or code assigned to each individual student. These 
identifiers enable states to track students over time and to 
link with other state databases, such as those maintained 
by postsecondary institutions or state employment agen- 
cies. Twenty-four states currently have statewide student 
identifiers. 2 

State education departments will need to collect student- 
level data on state test scores, fall enrollment, subsidized 
school lunch eligibility, limited English proficiency, racial or 
ethnic background, and special education status, as well 


The key elements of a comprehensive 
state student data system 

According to the National Center for Educational Ac- 
countability (NCEA), whose mission is to help state lead- 
ers use data to monitor, analyze, and improve student 
and school performance, the following ten elements are 
essential for an adequate statewide data system: 

1 . Unique statewide student identifier. 

2. Student-level enrollment data, including demo- 
graphic characteristics and program participation. 

3. Student-level state test data amenable to disaggre- 
gation by individual test items and objectives. 

4. Information on untested students, including 
reasons why those students were not tested. 

5. Student-level course completion data. 

6. Student-level SAT, ACT, and Advanced Placement 
(AP) test results. 

7. Student-level graduation and dropout data. 

8. State data audit process to ensure the accuracy 
of data submitted by districts. 

9. Capacity to match K— 1 2 and postsecondary 
education data. 

10. State data audit system. 

For more information on these elements, see Measur- 
ing the K-16 Pipeline: Ten Essential Elements of State- 
wide Data Collection Systems on the NCEA website at 
www.nc4ea.org . 

as other student demographic and program participation 
information. These data have proven significant in previous 
research on student achievement and are the basic founda- 
tion of a comprehensive student data collection system. 


1 Smith, N. J. 2003. Benefits of Linking Student Records, Education Data 
Systems for No Child Left Behind Reporting. Council of Chief State 
School Officers, Chiefs' Summer Institute Pre-Conference Session, Lake 
Tahoe, NV, July 26, 2003. http://www.ccsso.org/whats new/3072 .cfm? 
printthispage=1& (accessed: October 28, 2004). 

2 Alabama, Alaska, Arizona, Arkansas, Colorado, Connecticut, Delaware, 
Florida, Georgia, Hawaii, Louisiana, Massachusetts, Michigan, Minnesota, 
Mississippi, Nevada, Oregon, South Dakota, Tennessee, Texas, Vermont, 
Washington, West Virginia, and Wyoming. 
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With longitudinal student data, educators 
and policymakers can better understand 
how their students are 
performing over time and 
how their schools are doing 
in addressing the academic 
needs of all students. 
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